Researchers from the UK and Switzerland have found a mathematical means of helping regulators and business police Artificial Intelligence systems' biases towards making unethical, and potentially very costly and damaging choices. The collaborators from the University of Warwick, Imperial College London, and EPFL – Lausanne, along with the strategy firm Sciteb Ltd, believe that in an environment in which decisions are increasingly made without human intervention, there is a very strong incentive to know under what circumstances AI systems might adopt an unethical strategy--and to find and reduce that risk, or eliminate entirely, if possible. Artificial intelligence (AI) is increasingly deployed in commercial situations. Consider for example using AI to set prices of insurance products to be sold to a particular customer. There are legitimate reasons for setting different prices for different people, but it may also be more profitable to make certain decisions that end up hurting the company.

It essentially boils down to the idea that "if there is an advantage to something that will be perceived as unethical, then it is quite likely the machine learning is going to find it," says Robert MacKay, a mathematician at the University of Warwick and an author of the paper. MacKay uses the example of an algorithm that prices insurance products. If it is optimized only to maximize revenue, it's likely to treat customers unfairly and even unlawfully, selecting a higher price for users whose names code as non-white. In their paper, MacKay and his colleagues lay out complex mathematics that can help businesses and regulators detect the unethical strategies an algorithm might pursue in a given space and identify how the AI should be modified to prevent that behavior. It essentially boils down to the idea that "if there is an advantage to something that will be perceived as unethical, then it is quite likely the machine learning is going to find it," says Robert MacKay, a mathematician at the University of Warwick and an author of the paper.

If an artificial intelligence aims to maximise risk-adjusted return, then under mild conditions it is disproportionately likely to pick an unethical strategy unless the objective function allows sufficiently for this risk. Even if the proportion ${\eta}$ of available unethical strategies is small, the probability ${p_U}$ of picking an unethical strategy can become large; indeed unless returns are fat-tailed ${p_U}$ tends to unity as the strategy space becomes large. We define an Unethical Odds Ratio Upsilon (${\Upsilon}$) that allows us to calculate ${p_U}$ from ${\eta}$, and we derive a simple formula for the limit of ${\Upsilon}$ as the strategy space becomes large. We give an algorithm for estimating ${\Upsilon}$ and ${p_U}$ in finite cases and discuss how to deal with infinite strategy spaces. We show how this principle can be used to help detect unethical strategies and to estimate ${\eta}$. Finally we sketch some policy implications of this work.